Image processing apparatus, image processing method, and non-transitory computer-readable storage medium

ABSTRACT

A color temperature of each of a first sensed image that is sensed by a first image sensing device and a second sensed image that is sensed by a second image sensing device different from the first image sensing device is acquired, a color temperature that is common between the first image sensing device and the second image sensing device is decided based on the acquired color temperatures, and color information in an image sensed by the first image sensing device and an image sensed by the second image sensing device is adjusted based on the decided color temperature.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/149,670, filed Oct. 2, 2018, which is a continuation of U.S. Pat. No.10,122,980, filed Feb. 24, 2016, which claims the benefit of andpriority to Japanese Patent Application Nos. 2015-040687, filed Mar. 2,2015 and 2015-250498, filed Dec. 22, 2015, each of which are herebyincorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique for adjusting a colorbalance between a plurality of cameras.

Description of the Related Art

In recent years a so-called MR (Mixed Reality) technique is known as atechnique for causing the real world and a virtual world to beseamlessly blended in real-time. One known MR technique is a techniquein which a video see-through HMD (Head-Mounted Display) is used, anobject that approximately matches an object observed from a pupilposition of a HMD apparatus user is sensed by a video camera or thelike, and the HMD apparatus user can observe an image in which CG(Computer Graphics) is superimposed on the sensed image.

Japanese Patent Laid-Open No. 2004-205711 discloses, as an MR systemthat uses a video see-through type HMD, a technique that uses an HMDprovided with a first camera that senses a field-of-view region of a HMDuser, and a second camera that senses an image in order to detect aposition and orientation of the HMD.

Japanese Patent No. 4522307 discloses, as an MR system that uses a videosee-through type HMD, a technique for causing brightness or color inleft and right sensed images from sensed luminance information to matchamong a plurality of cameras for a right eye and for a left eye.

However, the techniques disclosed in the above-described patentliterature have a problem as below.

Japanese Patent No. 4522307 discloses using a common luminance value,obtained in accordance with processing for averaging luminance from aplurality of differing pieces of luminance information sensed by aplurality of cameras, to perform WB (white balancing) correction controlin a unified manner. However, in the technique disclosed in JapanesePatent No. 4522307, if a characteristic, a setting, or the like amongthe plurality of cameras exceeds an allowable range, it is not possibleto perform unified WB correction control in which color matches amongthe plurality of cameras.

Japanese Patent Laid-Open No. 2004-205711 is also similar; in aconfiguration in which cameras (image sensors) of differing types areequipped as is disclosed by Japanese Patent Laid-Open No. 2004-205711,there is a problem in that, for reasons such as the image sensors beingdifferent or luminance adjustment processing being performed inaccordance with a purpose where the purpose differs, it is not possibleto hold a common luminance value between a plurality of cameras, andunified WB correction control is not possible.

SUMMARY OF THE INVENTION

The present invention was conceived in view of these kinds of problems,and provides a technique for performing unified color balance adjustmentin which color matches among a plurality of cameras.

According to the first aspect of the present invention, there isprovided an image processing apparatus comprising: an acquisition unitconfigured to acquire a color temperature of each of a first sensedimage that is sensed by a first image sensing device and a second sensedimage that is sensed by a second image sensing device different from thefirst image sensing device; a decision unit configured to decide a colortemperature that is common between the first image sensing device andthe second image sensing device based on the color temperatures acquiredby the acquisition unit; and an adjustment unit configured to adjustcolor information in an image sensed by the first image sensing deviceand an image sensed by the second image sensing device based on thecolor temperature decided by the decision unit.

According to the second aspect of the present invention, there isprovided an image processing method comprising: acquiring a colortemperature of each of a first sensed image that is sensed by a firstimage sensing device and a second sensed image that is sensed by asecond image sensing device different from the first image sensingdevice; deciding a color temperature that is common between the firstimage sensing device and the second image sensing device based on theacquired color temperatures; and adjusting color information in an imagesensed by the first image sensing device and an image sensed by thesecond image sensing device based on the decided color temperature.

According to the third aspect of the present invention, there isprovided a non-transitory computer-readable storage medium storing acomputer program for causing a computer to function as an acquisitionunit configured to acquire a color temperature of each of a first sensedimage that is sensed by a first image sensing device and a second sensedimage that is sensed by a second image sensing device different from thefirst image sensing device; a decision unit configured to decide a colortemperature that is common between the first image sensing device andthe second image sensing device based on the color temperatures acquiredby the acquisition unit; and an adjustment unit configured to adjustcolor information in an image sensed by the first image sensing deviceand an image sensed by the second image sensing device based on thecolor temperature decided by the decision unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of processing that a head-mounted display 200performs.

FIG. 2 is a view for illustrating an example configuration of an MRsystem.

FIG. 3 is a view illustrating a principal configuration example in thehead-mounted display 200.

FIGS. 4A-4C are views for explaining the configuration of thehead-mounted display 200.

FIG. 5 is a block diagram for illustrating a more detailed exampleconfiguration of an image processing unit 113.

FIG. 6 is a view for illustrating natural features in an image, andfeature points thereof.

FIGS. 7A and 7B are flowcharts of processing of step S101 to step S104,and step S161 to step S164.

FIG. 8 is a view for illustrating an example configuration of a colortemperature detection table.

FIG. 9 is a view for illustrating colors at color temperatures.

FIGS. 10A and 10B are views for illustrating example configurations of atable referred to in step S190.

FIG. 11 is a view for illustrating an example configuration of thehead-mounted display 200 according to a first variation of the firstembodiment.

FIG. 12 is a view for illustrating an example configuration of thehead-mounted display 200 according to a second embodiment.

FIG. 13 is a flowchart of processing that the head-mounted display 200performs.

FIGS. 14A and 14B are views for explaining a method that decides aphenomenon in which detection precision of a color temperature varies,and a priority camera.

FIG. 15 overlaps color temperature detection tables 1501 and 1502 onFIG. 14B.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described hereinafter indetail, with reference to the accompanying drawings. Note thatembodiments described below merely illustrate examples of specificallyimplementing the present invention, and are only specific embodiments ofa configuration defined in the scope of the claims.

First Embodiment

Below, explanation is given of an example of an image processingapparatus that acquires a color temperature of each of a first sensedimage that is sensed by a first image sensing device and a second sensedimage that is sensed by a second image sensing device different from thefirst image sensing device, decides a color temperature that is commonbetween the first image sensing device and the second image sensingdevice based on the acquired color temperatures, and adjusts colorinformation in an image sensed by the first image sensing device and animage sensed by the second image sensing device based on the decidedcolor temperature. In this example, although explanation is given of acase in which the image processing apparatus is a head-mounted displaysuch as an HMD, the image processing apparatus that has a configurationsuch as this is not limited to being applied to a head-mounted display,and, for example, may be applied to a 3D display apparatus thatdisplays, on one screen, images for a right eye and for a left eye in astriped form.

Firstly, FIG. 2 is used to explain an example configuration of an MRsystem that includes a head-mounted display according to the presentembodiment. As illustrated in FIG. 2, the MR system has a head-mounteddisplay 200 that provides an image of a mixed reality space, which is aspace that blends a virtual space and a physical space in front of auser's eyes; a computing device 250 that generates the image of themixed reality space and provides it to the head-mounted display 200; anda cable 240 that connects the head-mounted display 200 and the computingdevice 250. Note that although the cable 240 is illustrated as a wiredcommunication path, a wireless communication path may be used instead.

Next, the configuration of the head-mounted display 200 will beexplained in further detail. The head-mounted display 200 is a so-calledvideo see-through type head-mounted display, and has sub-cameras (asub-camera-for-right-eye 21R and a sub-camera-for-left-eye 21L) thatsense images that are used to obtain a position and orientation of thehead-mounted display 200, and main cameras (a main-camera-for-right-eye20R and a main-camera-for-left-eye 20L) that sense images of a physicalspace that are composed with images of the virtual space when generatingan image of the mixed reality space.

The sub-cameras are cameras for sensing markers 210 disposed in aphysical space, and the computing device 250 can calculate a positionand orientation of the head-mounted display 200 by executingconventional processing that uses an image of a marker 210 sensed by asub-camera. Strictly speaking, in addition to using the image sensed bythe sub-camera-for-left-eye 21L to calculate the position andorientation of the main-camera-for-left-eye 20L, the image sensed by thesub-camera-for-right-eye 21R is used to calculate the position andorientation of the main-camera-for-right-eye 20R. The computing device250 generates an image of the mixed reality space by generating, basedon position and orientation of the main cameras, an image of the virtualspace from the perspective of the position and orientation of the maincameras, and composing the generated image of the virtual space with asensed image of the physical space by the main camera. Strictlyspeaking, by generating, based on a position and orientation of themain-camera-for-left-eye 20L, an image of the virtual space from theperspective of the position and orientation of themain-camera-for-left-eye 20L, and composing the generated image of thevirtual space with the sensed image of the physical space by themain-camera-for-left-eye 20L, an image of a mixed reality space that isprovided to a left eye of a user is generated. Also, by generating,based on a position and orientation of the main-camera-for-right-eye20R, an image of the virtual space from the perspective of the positionand orientation of the main-camera-for-right-eye 20R, and composing thegenerated image of the virtual space with the sensed image of thephysical space by the main-camera-for-right-eye 20R, an image of a mixedreality space that is provided to a right eye of a user is generated.The computing device 250 then sends the generated images of the mixedreality space (the image of the mixed reality space for the right eyeand the image of the mixed reality space for the left eye) to thehead-mounted display 200.

In this way, a purpose of the main cameras and the sub-cameras differs.As illustrated in FIG. 2, because the main camera has as an objectivesensing an image of the physical space to provide to the user (sensing auser's field-of-view region), an image sensor that is used is an imagesensor of a type (hypothetically defined as an X type) having featuressuch as having a broad field of view in a leftward/rightwardorientation, low noise, and a wide color gamut. In the presentembodiment, a type of a rolling shutter method is used as the X type. Incontrast, because the sub-cameras have an objective of sensing imagesused to obtain a position and orientation, they use image sensors of atype (hypothetically defined as a Y type) of a global shutter method,for which a field of view in an upward/downward orientation wheremarkers are present is wide, and moving object distortion of themarkers, called image flow, is not generated.

Note that, the number of sub-cameras may be 2 or more to expand an imagesensing range and increase a possibility of sensing a marker; also, thenumber of sub-cameras may be 1 to lower costs though detection precisionwill become lower, or conversely if it is a high-capability camerahaving a broad field of view in a leftward/rightward orientation aswell.

A principal example configuration of the head-mounted display 200 isillustrated in FIG. 3. Because there is a pair of configurations—for theright eye and for the left eye—only one is shown graphically in FIG. 3.The head-mounted display 200 has a display element 121, which iscomprised by small-scale liquid crystal displays for the right eye andfor the left eye; a display optical system 320 such as a free curvatureprism for performing a magnified display of the images for the right eyeand the left eye that are displayed on the display element 121; a maincamera 20 for sensing an object that approximately matches an objectobserved from a position of a pupil 300 of a head-mounted display 200wearer; and an imaging optical system 310 for causing the position ofthe pupil 300 to approximately match the position of the main camera 20.The sub-camera (not shown) is arranged on the outside of the main camera20 with respect to a face center of the head-mounted display 200 wearer.

Next, the block diagram of FIG. 4A is used to explain the hardwareconfiguration example of the head-mounted display 200.

A right-eye main sensing unit 100R is a capturing unit that includes amain-camera-for-right-eye 20R, an optical system of themain-camera-for-right-eye 20R, and various processing circuits for themain-camera-for-right-eye 20R.

A left-eye main sensing unit 100L is a capturing unit that includes themain-camera-for-left-eye 20L, an optical system of themain-camera-for-left-eye 20L, and various processing circuits for themain-camera-for-left-eye 20L.

A right sub sensing unit 110R is a capturing unit that includes thesub-camera-for-right-eye 21R, an optical system of thesub-camera-for-right-eye 21R, and various processing circuits for thesub-camera-for-right-eye 21R.

A left sub sensing unit 110L is a capturing unit that includes thesub-camera-for-left-eye 21L, an optical system of thesub-camera-for-left-eye 21L, and various processing circuits for thesub-camera-for-left-eye 21L.

A display-unit-for-right-eye 120R is attached onto the head-mounteddisplay 200 so as to be positioned in front of a right eye of a userwearing the head-mounted display 200, and displays an image, generatedby the computing device 250, of the mixed reality space for the righteye.

A display-unit-for-left-eye 120L is attached onto the head-mounteddisplay 200 so as to be positioned in front of a left eye of a userwearing the head-mounted display 200, and displays an image, generatedby the side of the computing device 250, of the mixed reality space forthe left eye.

By executing processing that uses data or a computer program stored in amemory 131 of the head-mounted display 200, a MPU 130 performs operationcontrol of each above-described functional unit that is connected to abus 190, and also performs operation control of the head-mounted display200 as a whole.

The memory 131 includes various memories, such as a memory for storinginformation, which is explained as known information in the followingexplanation, and data or a computer program for causing the MPU 130 toexecute each process that is later explained as something that the MPU130 performs; a memory that has a work area used when the MPU 130executes various processing; or the like.

An example configuration of a capturing unit that can be applied to eachof the right-eye main sensing unit 100R, the left-eye main sensing unit100L, the right sub sensing unit 110R, and the left sub sensing unit110L is illustrated in FIG. 4B.

An image sensor 111 is a CCD image sensor or the like, converts light inthe external world into an analog electrical signal and outputs it, andis driven in accordance with signals from a TG (a timing generator) 115and a V-Dr (V driver) 116 that generates a signal of a verticaldirection after receiving a signal from the TG 115.

An A/D converter 112 converts an analog electrical signal output fromthe image sensor 111 to a digital electrical signal.

By applying various image processing to the digital electrical signalconverted by the A/D converter 112, the image processing unit 113generates and outputs a sensed image that has been image processed. Thisimage processing includes processing for adjusting a color balance thatis described later.

A sensed image output unit 114 outputs the sensed image, to which theimage processing by the image processing unit 113 has been applied, tothe computing device 250 in an appropriate image format. Note that anoutput destination of the sensed image output unit 114 is not limited tothe computing device 250.

An example configuration of a display unit that can be applied to eachof the display-unit-for-right-eye 120R and the display-unit-for-left-eye120L is illustrated in FIG. 4C.

A display image input unit 123 receives an image of a mixed realityspace that is output from the computing device 250, and transfers thereceived image of the mixed reality space to a display driving unit 122.

The display driving unit 122 drives the display element 121 to cause theimage of the mixed reality space transferred from the display imageinput unit 123 to be displayed.

The display element 121 is a display element, such as p-Si TFT or LCOS,is driven by the display driving unit 122, and displays the image of themixed reality space that the display image input unit 123 receives fromthe computing device 250.

Note that, as described above, the computing device 250 uses the imagesensed by the sub-camera to calculate the position and orientation ofthe main camera, but, in more detail, the computing device 250 detects amarker from the sensed image in accordance with image analysis, andacquires information such as a size, a shape, a fill pattern, or thelike of the detected marker. From this information acquired by themarker detection, the computing device 250 calculates three-dimensionalposition and orientation information regarding a relative positionalrelationship between the marker and the head-mounted display 200, and adirection in which a user who is wearing the head-mounted display 200observes the marker. In this way, by using a plurality of markers anddefining beforehand a positional relationship of each marker asindicator arrangement information, it becomes possible to calculate adirection in which a marker is observed from these relative positionalrelationships. Accordingly, rather than a marker by which discriminationof even a direction is possible by an internal fill pattern, it ispossible to use a marker that holds unidimentional information and doesnot hold directional information such as a light-emitting element (e.g.an LED), a color marker or the like, for example. In addition, insteadof the markers 210 as illustrated in FIG. 2, it is possible to extractnatural features in an image, such as for example outlines of a door600, a table 605, and a window 610 as in FIG. 6, specific colors in theimage, or the like, and use these to calculate three-dimensionalposition and orientation information. The reference numerals 650 and 655illustrate a portion of feature points of the door 600 by “x” marks. Byusing a plurality of markers of the same type, using several types ofmarkers simultaneously, or by combining and using information of featurepoints in the image and marker information, it is possible to generatethree-dimensional position and orientation information of a higherprecision.

By the above configuration, a viewer, by wearing the head-mounteddisplay 200 on his or her head, can experience a mixed reality spacethat seamlessly blends the physical space and the virtual space inreal-time.

Next, a block diagram of FIG. 5 is used to explain an exampleconfiguration of the image processing unit 113 in further detail.

By interpolation from adjacent pixels (adjacent pixels are pixels whosecolors are different to each other) in an image (an image of, forexample, a Bayer pattern in which each pixel is one of R, G, or B)expressed by a digital electrical signal output from the A/D converter112, a color interpolation unit 501 obtains (restores) pixel values ofeach of R, G, and B in each pixel that configures the sensed image.

A WB wave-detection unit 505 sets the entire sensed image, which isgenerated by the color interpolation unit 501, or a partial regionthereof (for example, a partial region of a prescribed size that iscentered on a center position of the sensed image) as a target region,and obtains average pixel values (an R average pixel value, a G averagepixel value, and a B average pixel value) in the target region. The Raverage pixel value in the target region is a result of obtaining atotal value of R component pixel values of all pixels included in thetarget region, and dividing the total value by the number of pixelsincluded in the target region. This is similar when obtaining the Gaverage pixel value and the B average pixel value. The WB wave-detectionunit 505 then sends the respective R, G, and B average pixel valuesobtained for the target region to the MPU 130.

A WB correction unit 502 receives a R WB correction value, a G WBcorrection value, and a B WB correction value that the MPU 130 decidedbased on the R average pixel value and the B average pixel value, and byperforming processing such as gain correction in accordance with theseWB correction values with respect to the sensed image generated by thecolor interpolation unit 501, the WB correction unit 502 adjusts anachromatic color balance, and with this realizes WB correction withrespect to the sensed image.

A color correction unit 503 corrects a color of a sensed image for whichthe color balance has been adjusted by the WB correction unit 502, and agamma correction unit 504 corrects a tone of a brightness of a sensedimage for which the color has been corrected by the color correctionunit 503.

Next, regarding processing that the head-mounted display 200 (mainly theimage processing unit 113 and the MPU 130) performs, explanation isgiven using FIG. 1, which illustrates a flowchart of the sameprocessing.

<Step S101>

By the right-eye main sensing unit 100R and the MPU 130 performingprocessing in accordance with the flowchart of FIG. 7A, a colortemperature in the entire image sensed by the right-eye main sensingunit 100R or a partial region thereof is acquired.

<Step S102>

By the left-eye main sensing unit 100L and the MPU 130 performingprocessing in accordance with the flowchart of FIG. 7A, a colortemperature in the entire image sensed by the left-eye main sensing unit100L or a partial region thereof is acquired.

<Step S103>

By the right sub sensing unit 110R and the MPU 130 performing processingin accordance with the flowchart of FIG. 7A, a color temperature in theentire image sensed by the right sub sensing unit 110R or a partialregion thereof is acquired.

<Step S104>

By the left sub sensing unit 110L and the MPU 130 performing processingin accordance with the flowchart of FIG. 7A, a color temperature in theentire image sensed by the left sub sensing unit 110L or a partialregion thereof is acquired.

Below, explanation is given for a case in which processing in accordancewith the flowchart of FIG. 7A is executed in step S101. In such a case,processing in accordance with the flowchart of FIG. 7A is executed bythe MPU 130 and functional units in the right-eye main sensing unit100R. Note that if executing processing in accordance with the flowchartof FIG. 7A in step S102, processing in accordance with the flowchart ofFIG. 7A is executed by the MPU 130 and functional unit in the left-eyemain sensing unit 100L. Also if executing processing in accordance withthe flowchart of FIG. 7A in step S103, processing in accordance with theflowchart of FIG. 7A is executed by the MPU 130 and functional unit inthe right sub sensing unit 110R. Also if executing processing inaccordance with the flowchart of FIG. 7A in step S104, processing inaccordance with the flowchart of FIG. 7A is executed by the MPU 130 andfunctional unit in the left sub sensing unit 110L.

<Step S180>

The WB wave-detection unit 505 sets the entire sensed image generated bythe color interpolation unit 501 or a partial region thereof as a targetregion, and obtains average pixel values (the R average pixel value, theG average pixel value, and the B average pixel value) of the targetregion. The WB wave-detection unit 505 then sends the respective R, G,and B average pixel values obtained for the target region to the MPU130.

<Step S181>

The MPU 130 uses the B average pixel value and the R average pixel valuereceived from the WB wave-detection unit 505 to obtain (the B averagepixel value/the R average pixel value) as a WB ratio.

<Step S182>

The MPU 130 uses a color temperature detection table generated for eachcapturing unit in advance to acquire a color temperature thatcorresponds to the WB ratio obtained in step S181. An exampleconfiguration of the color temperature detection table is illustrated inFIG. 8. The abscissa axis in FIG. 8 indicates the WB ratio and theordinate axis indicates the color temperature. Reference numeral 800represents a color temperature detection table that expresses acorrespondence relationship between the WB ratio and the colortemperature corresponding to the image sensor (X type) used by the maincameras, and reference numeral 801 represents a color temperaturedetection table that expresses the correspondence relationship betweenthe WB ratio and the color temperature corresponding to an image sensor(Y type) used by the sub-cameras. For the color temperature conversiontables, because a characteristic differences according to each type ofimage sensor is large compared to left and right individual differencesof image sensors of the same type, it is assumed that the left and rightindividual differences can been ignored, and that color temperaturedetection table are only registered according to type. In other words,in step S101 and step S102, the common color temperature detection table800 is used to acquire respective color temperatures, and in step S103and step S104, the common color temperature detection table 801 is usedto acquire respective color temperatures.

In this way, by executing processing in accordance with the flowchart ofFIG. 7A in step S101 to in step S104, a color temperature T1 in theimage sensed by the right-eye main sensing unit 100R or a partial regionthereof, a color temperature T2 in the image sensed by the left-eye mainsensing unit 100L or a partial region thereof, a color temperature T3 inthe image sensed by the right sub sensing unit 110R or a partial regionthereof, and a color temperature T4 in the image sensed by the left subsensing unit 110L or a partial region thereof can be acquired.

<Step S110>

The MPU 130 obtains a maximum difference (color temperature difference)ΔT, among T1, T2, T3, and T4. This is firstly obtaining Tmax=MAX (T1,T2, T3, T4), and Tmin=MIN (T1, T2, T3, T4). Here, MAX(x1, x2, . . . ,xn) is a function that returns a maximum value from x1 to xn, andMIN(x1, x2, . . . , xn) is a function that returns a minimum value fromx1 to xn. ΔT=Tmax−Tmin is then calculated.

For example, if T1=3500K, T2=3550K, T3=3650K, and T4=3900K (a unit K ofcolor temperature is Kelvin), the color temperature Tmax=MAX(T1, T2, T3,T4)=3900K, and the color temperature Tmin=MIN(T1, T2, T3, T4)=3500K, sothe color temperature difference ΔT=400K.

Additionally, explanation is given for a reason as to why the colortemperatures acquired from the sensed image by each capturing unit varyin this way. Firstly, a difference in actual color temperature among thecapturing units may occur due to rays of ambient light input beingdifferent due to differing attachment positions of the capturing units(20R, 20L, 21R, 21L) as illustrated in FIG. 2. In particular, thesub-cameras are arranged on the left and right outside of the maincameras, and moreover have a field of view that is wider in theupward/downward orientation; an image center orientation is also shiftedin the upward/downward orientation, or the like, and ambient light isfocused differently to the main camera. This is because there are casesin which light entering a capturing unit differs depending on theorientation or location in which various ambient light, such as light inaccordance with a ceiling light, indirect light where light of theceiling light has reflected from a wall, sunlight entering from awindow, or the like, mixes. In addition, as another reason why colortemperature varies, there are also individual differences of sensors ina capturing unit.

<Step S120>

The MPU 130 determines whether the color temperature difference ΔTobtained in step S110 is within the allowable range ΔTerror (in anallowable range). For example, it is determined whether the absolutevalue of the color temperature difference ΔT is less than or equal tothe allowable range ΔTerror.

The allowable range ΔTerror is for example a value such as 500K. Thevalue of the allowable range ΔTerror is set in advance from a detectionerror that occurs due to individual differences of image sensors in acapturing unit, a condition of entrance of rays of ambient lightaccording to an attachment position of a capturing unit, or the like.Note that a value of the allowable range ΔTerror is not limited to afixed value, and may be a fluctuating value for which an average colortemperature of all capturing units or an error allowed in accordancewith a color temperature of a specific capturing unit differs.

As a result of this determination, if the color temperature differenceΔT is within the allowable range ΔTerror, the processing proceeds tostep S130, and if the color temperature difference ΔT is not within theallowable range ΔTerror, the processing proceeds to step S140.

<Step S130>

The MPU 130 decides the average color temperature Tave of T1, T2, T3,and T4 as the common color temperature of the right-eye main sensingunit 100R, the left-eye main sensing unit 100L, the right sub sensingunit 110R, and the left sub sensing unit 110L. The average colortemperature Tave can be calculated in accordance with the followingformula.Tave=AVE(T1,T2,T3,T4)

Here AVE(x1, x2, . . . , xn) is a function that returns an average valueof x1 through to xn. For example, if T1=3500K, T2=3550K, T3=3650K, andT4=3900K, then the average color temperature Tave=3650K.

<Step S140>

As the color temperature difference ΔT exceeds the allowable rangeΔTerror and it is difficult to handle the color temperature of allcapturing units the same, the MPU 130 selects, in accordance with aprescribed standard, one of T1, T2, T3, and T4 as the common colortemperature among the right-eye main sensing unit 100R, the left-eyemain sensing unit 100L, the right sub sensing unit 110R, and the leftsub sensing unit 110L. In the present embodiment, a standard colortemperature is determined in advance, and the color temperature closestto the standard color temperature from T1, T2, T3, and T4 is selected asthe common color temperature among the right-eye main sensing unit 100R,the left-eye main sensing unit 100L, the right sub sensing unit 110R,and the left sub sensing unit 110L. For example, if a standard lightsource as D55 and a standard color temperature of 5500K is assumed, andT1=3500K, T2=3550K, T3=3650K, and T4=3900K, then because the colortemperature closest to the standard color temperature from T1, T2, T3,and T4 is T4 (3900K), then T4 is selected as the common colortemperature among the right-eye main sensing unit 100R, the left-eyemain sensing unit 100L, the right sub sensing unit 110R, and the leftsub sensing unit 110L.

Here, explanation is given for a reason why the color temperatureclosest to the standard color temperature is selected. Firstly, a reasonwhy a standard ambient light is arranged is that since a design in whichmaintaining WB precision maximally is difficult for any colortemperature among a plurality of color temperatures, a color temperaturethat should be often used in ambient light when using the head-mounteddisplay 200 is set so as to achieve a design in which WB precisionbecomes highest by that color temperature. Because a color balance forwhen WB correction is performed at a time of a standard colortemperature 5500K is a standard for a design, it means that regardlessof what the color temperature environment is, all approach a WBcondition at a time of the standard color temperature 5500K after WBcorrection.

For example, if T1=3500K, T2=3550K, T3=3650K, and T4=3900K, a value of awarmer color than the standard color temperature 5500K will is outputfor all color temperatures, and it can be seen from the fact that theaverage color temperature Tave=3650K calculated in step S130 is a yellowthat is towards red. FIG. 9 is a view for illustrating colors at colortemperatures. Here, if the average color temperature Tave=3650K is acorrect value as the actual ambient light, after WB correction, itclosely matches the WB condition at a time of the standard colortemperature 5500K as intended. However, if the detected average colortemperature Tave=3650K is a value shifted to be redder than the actualambient light, there is an overcorrection when WB correction isperformed, which results in a WB condition shifted to a blue which is acold color. At this time, when a WB corrected image becomes of coldcolors even though the actual ambient light is of warm colors, a problemoccurs in that as a characteristic of visual perception in humans, anerror in the correction is recognized as large. Accordingly, so not tocause this kind of overcorrection in a WB correction to be occur, it ispossible set a shifted value of a warm color side, which is the actualambient light which is close to the WB condition for the time of thestandard color temperature 5500K, by performing WB correction afterselecting a color temperature close to the standard color temperature5500K if the color temperature difference is large. For the abovereason, even if a shift amount of the WB condition of a time of thestandard color temperature 5500K becomes large, if it is on the warmcolor side, which is the actual ambient light, the correction error isrecognized as being smaller than a time of an overcorrection to a coldcolor in WB correction, and therefore in step S140 processing aims forthis effect. Note that, although the effect of and reason for step S140are explained with a case in which the detected color temperature is ofa warm color, even if the detected color temperature is oppositely of acold color that is higher than the standard color temperature 5500K, itis possible to avoid a problem due to overcorrection for the samereason.

<Step S150>

The MPU 130 sets the color temperature determined in step S130 or thecolor temperature selected in step S140 as the color temperature that iscommon (common color temperature) among the right-eye main sensing unit100R, the left-eye main sensing unit 100L, the right sub sensing unit110R, and the left sub sensing unit 110L.

Here, explanation is given for a reason why among all capturing units acommon color temperature is set and unified WB correction is performed.As previously explained, a reason that the color temperature of aplurality of capturing units differs is that in the case of a detectionerror due to detection of ambient light that differs in accordance withan arrangement of the capturing unit, due to individual differences ofthe image sensors, or the like, if a common color temperature is notused, and color temperatures detected by each capturing unit are subjectto WB correction, a problem occurs in that the sensed images of therespective capturing units will have different color. In particular, aproblem occurs when, in a case of WB detection at a timing chosen by auser, called one-push rather than WB control, called AWB, in which WB isautomatically detected in short fixed intervals, a user wearing thehead-mounted display 200 moves from an area where WB detection wasperformed, the color differs between capturing units, causing the userto have a significant sense of unnaturalness. For reasons such as theabove, it is necessary to use a common color temperature among allcapturing units to perform unified WB correction. In addition, a reasonthat a color temperature is used for unified WB correction among allcapturing units is to handle a case in which a luminance valuecharacteristic is very different between capturing units, such as wheretypes of image sensors are different. In other words, by respectivelyapplying reverse characteristics to calculate a color temperature ofambient light from luminance values for which characteristics differ foreach capturing unit in wave-detection of ambient light, this colortemperature becomes an indicator that can be commonly used between thecapturing units.

<Step S161>

By performing processing in accordance with the flowchart of FIG. 7B,the MPU 130 uses the common color temperature to obtain a WB correctionvalue to be used in WB correction towards an image sensed by theright-eye main sensing unit 100R, and sends the obtained WB correctionvalue to the right-eye main sensing unit 100R.

<Step S162>

By performing processing in accordance with the flowchart of FIG. 7B,the MPU 130 uses the common color temperature to obtain a WB correctionvalue to be used in WB correction towards an image sensed by theleft-eye main sensing unit 100L, and sends the obtained WB correctionvalue to the left-eye main sensing unit 100L.

<Step S163>

By performing processing in accordance with the flowchart of FIG. 7B,the MPU 130 uses the common color temperature to obtain a WB correctionvalue to be used in WB correction on an image sensed by the right subsensing unit 110R, and sends the obtained WB correction value to theright sub sensing unit 110R.

<Step S164>

By performing processing in accordance with the flowchart of FIG. 7B,the MPU 130 uses the common color temperature to obtain a WB correctionvalue to be used in WB correction on an image sensed by the left subsensing unit 110L, and sends the obtained WB correction value to theleft sub sensing unit 110L.

Below, explanation is given for a case in which processing in accordancewith the flowchart of FIG. 7B is executed in step S161. In such a case,the MPU 130 obtains a WB correction value for the right-eye main sensingunit 100R, and sends it to the right-eye main sensing unit 100R.Additionally, when executing processing in accordance with the flowchartof FIG. 7B in step S162, the MPU 130 obtains a WB correction value forthe left-eye main sensing unit 100L and sends it to the left-eye mainsensing unit 100L. Additionally, when executing processing in accordancewith the flowchart of FIG. 7B in step S163, the MPU 130 obtains a WBcorrection value for the right sub sensing unit 110R and sends it to theright sub sensing unit 110R. Additionally, when executing processing inaccordance with the flowchart of FIG. 7B in step S164, the MPU 130obtains a WB correction value for the left sub sensing unit 110L andsends it to the left sub sensing unit 110L.

<Step S190>

The MPU 130 obtains the R WB correction value, the G WB correctionvalue, and the B WB correction value from the common color temperature.At that time, the MPU 130 refers to a table exemplified in FIGS. 10A and10B. In FIGS. 10A and 10B, the abscissa axis indicates colortemperature, and the ordinate axis indicates a correction gain value (WBcorrection value). The table of FIG. 10A is a table corresponding to theimage sensors (X type) used in the main cameras, and the table of FIG.10B is a table corresponding to the image sensors (Y type) used in thesub-cameras. Here when executing processing in accordance with of theflowchart of FIG. 7B in step S161 and step S162, the table of FIG. 10Ais referred to, and when executing processing in accordance with theflowchart of FIG. 7B in step S163 and step S164, the table of FIG. 10Bis referred to.

In FIGS. 10A and 10B, RGB correction gain values are, with G as astandard, R/G (reference numerals 1010 and 1050) and B/G (referencenumerals 1020 and 1060), which correspond to the abscissa axis colortemperature. However, for example if a color temperature T=3650K, thecorrection gain value in the image sensor (X type) becomes (R, G,B)=(0.9252, 1.0, 1.9633) by the table of FIG. 10A, and the correctiongain value in the image sensor (Y type) is calculated as (R, G,B)=(0.7857, 1.0, 1.5678) by the table of FIG. 10B.

In this way, the tables of FIG. 10A or FIG. 10B are referred to, R/G andB/G which correspond to the common color temperature are specified, andR and B values for when G=1.0 are specified. The values of R, G, and Bthus specified in this way respectively become the R WB correctionvalue, the G WB correction value, and the B WB correction value.

<Step S192>

The MPU 130 sends the R WB correction value, the G WB correction value,and the B WB correction value obtained in step S190 to the WB correctionunit 502 which the image processing unit 113 in the right-eye mainsensing unit 100R has. With this, the WB correction unit 502 receivesthe R WB correction value, the G WB correction value, and the B WBcorrection value received from the MPU 130, and by performing processingsuch as gain correction in accordance with these WB correction valueswith respect to the sensed image generated by the color interpolationunit 501, achromatic color balance is adjusted, and with this WBcorrection on the sensed image is realized.

In this way, by WB control processing according to the presentembodiment, it is possible to perform unified WB correction on sensedimages of each of the main cameras and the sub-cameras.

Note that, in the present embodiment, although average pixel values ofeach of R, G, and B are obtained and sent to the MPU 130 in step S180,in step S181, the B average pixel value and the R average pixel valueare used to perform processing, and because the G average pixel value isnot used, configuration may be taken to not obtain the G average pixelvalue.

In the present embodiment, although explanation was given for a case inwhich the color components of an image are an R component, a Gcomponent, and a B component, other color components may be used, andfor example, the present embodiment can be similarly applied to colorcomponents in a Lab color space as targets.

In addition, in the present embodiment, although configuration was takento perform WB correction on each sensed image of the main cameras andthe sub-cameras, configuration may be taken to perform WB correction foran image provided to an eye of a user—in other words an image sensed bya main camera, and not perform WB correction for an image not providedto an eye of a user—in other words an image sensed by a sub-camera.

First Variation

In the first embodiment, the head-mounted display 200 has a total ofthree or more cameras: two main cameras (a camera for a right eye and acamera for a left eye) for sensing a field-of-view region of a user whowears the head-mounted display 200 on their head, and one or moresub-cameras that sense an image used for detecting the a position andorientation. However, as illustrated in FIG. 11, configuration may betaken to have two cameras: a camera 22R that serves both the purpose ofthe right-eye main sensing unit 100R and the purpose of the right subsensing unit 110R, and a camera 22L that serves both the purpose of theleft-eye main sensing unit 100L and the purpose of the left sub sensingunit 110L.

In such a case, an image sensed by the camera 22R is used as an imagethat is composed with an image of a virtual space when generating animage of a mixed reality space presented to a right eye of a user thatwears the head-mounted display 200, and is also used for detecting aposition and orientation of the camera 22R. Also, an image sensed by thecamera 22L is used as an image that is composed with an image of avirtual space when generating an image of a mixed reality spacepresented to a left eye of a user that wears the head-mounted display200, and is also used for detecting a position and orientation of thecamera 22L.

In the case of this kind of the head-mounted display 200, it is possibleto acquire, by processing similar to the above-described processing, acommon color temperature by the camera 22R and the camera 22L, obtain,from the acquired common color temperature, a WB correction value forthe camera 22R and a WB correction value for the camera 22L, use the WBcorrection value for the camera 22R to perform WB correction on theimage sensed by the camera 22R, and also use the WB correction value forthe camera 22L to perform WB correction on the image sensed by thecamera 22L.

Second Variation

In the first embodiment, the head-mounted display 200, as somethinghaving a total of 4 cameras: 2 main cameras (the camera for a right eyeand the camera for a left eye) for sensing a field-of-view region of auser who wears the head-mounted display 200 on their head, and 2sub-cameras (the right sub-camera and the left sub-camera) that sense animage used for detecting the a position and orientation, calculates acolor temperature in a configuration in which differing image sensortypes are mixed such as an averaging calculation. However, configurationmay be taken to calculate common color temperatures dividing by imagesensors in combinations of the same type, and to use the calculatedcommon color temperatures divided for each type to perform common WBcorrection for each type of camera. In other words, WB correctioncontrol of the present embodiment may be performed between a pluralityof cameras having image sensors of the same type, and independently foreach type of image sensor.

Third Variation

In the first embodiment, if the color temperature difference ΔT iswithin the allowable range ΔTerror, the processing proceeds to stepS130, and if the color temperature difference ΔT is not within theallowable range ΔTerror, the processing proceeds to step S140. However,configuration may be such that if the color temperature difference ΔT iswithin the allowable range ΔTerror, the processing proceeds to stepS140, and if the color temperature difference ΔT is not within theallowable range ΔTerror, the processing proceeds to step S130. With sucha configuration, it is effective in a case such as where the error in WBcorrection becomes small.

Fourth Variation

In the first embodiment, in step S140 a color temperature, among of T1,T2, T3, and T4, that is closest to the standard color temperature isselected as a common color temperature among the right-eye main sensingunit 100R, the left-eye main sensing unit 100L, the right sub sensingunit 110R, and the left sub sensing unit 110L. However, in step S140,because it is good if a preferred color temperature can be selected,various methods can be considered for a method that decides a commoncolor temperature in accordance with each capturing unit.

For example, among T1, T2, T3, and T4, a color temperature closest to amedian value thereof may be selected. In other words, configuration maybe taken to select one based on a statistic obtained from T1, T2, T3,and T4. In addition, configuration may be taken to select a colortemperature to be a predetermined value for the head-mounted display200.

Fifth Variation

In the first embodiment, in step S130 an average color temperature Taveof all (the entirety) of T1, T2, T3, and T4 is decided as a common colortemperature among the right-eye main sensing unit 100R, the left-eyemain sensing unit 100L, the right sub sensing unit 110R, and the leftsub sensing unit 110L. However, instead of all of T1, T2, T3, and T4, anaverage color temperature of a portion thereof may be taken as Tave. Forexample, configuration may be taken to obtain an average value of all ofT1, T2, T3, and T4, to remove the color temperature among T1, T2, T3,and T4 for which the difference with the average value is largest, andto take an average value of the remaining color temperatures as Tave.

Second Embodiment

In the first embodiment, explanation was given of as something in whichthe head-mounted display 200 has capturing unit groups in which imagesensors are of different types, but the head-mounted display 200 mayhave a capturing unit group in which image sensors are of the same type.In such a configuration, a difference in color temperature detectionresults between capturing units due to individual differences of imagesensors becomes a main problem. Thus, in the present embodiment, WBcorrection control for reducing WB correction error between capturingunits due to individual differences of image sensors is performed.

Below, differences with the first embodiment are predominantlydescribed, and to the extent that something is not particularly touchedon below, it is similar to in the first embodiment.

The head-mounted display 200 according to the present embodiment, asillustrated in FIG. 12, has two cameras: a camera 23R that serves boththe purpose of the right-eye main sensing unit 100R and the purpose ofthe right sub sensing unit 110R, and a camera 23L that serves both thepurpose of the left-eye main sensing unit 100L and the purpose of theleft sub sensing unit 110L, and in addition an image sensor is the sametype for both cameras (Y type).

In this kind of the head-mounted display 200, instead of processing inaccordance with the flowchart of FIG. 1, processing in accordance with aflowchart of FIG. 13 is performed. Processing steps in FIG. 13 that arethe same as processing steps illustrated in FIG. 1 have the same stepnumber added thereto, and explanation corresponding to such processingsteps is omitted or simplified.

<Step S201>

By performing processing in accordance with the flowchart of FIG. 7A,the camera 23R and the MPU 130 acquire the color temperature T1 in anentire image sensed by the camera 23R or a partial region thereof.

<Step S202>

By performing processing in accordance with the flowchart of FIG. 7A,the camera 23L and the MPU 130 acquire the color temperature T2 in anentire image sensed by the camera 23L or a partial region thereof.

In step S110, whereas in the first embodiment four color temperatures(T1, T2, T3, and T4) were made to be targets, in the present embodiment,because there are only two: T1 and T2, ΔT is simply obtained bycalculating ΔT=|T2−T1|. For example, if the color temperature T1 of thecamera 23R=3600K and the color temperature T2 of the camera 23L=3950K,then the color temperature difference ΔT is 350K. In this way, a causeof a difference in color temperature between the left and right camerasbeing generated is individual differences in the cameras due tocapability variation of the image sensors, an alignment between an imagesensor and a camera lens (mechanical tolerance), or the like.

In step S120, if the color temperature difference ΔT is within theallowable range ΔTerror, similarly to the first embodiment theprocessing proceeds to step S130, but if the color temperaturedifference ΔT is not within the allowable range ΔTerror, the processingproceeds to step S240.

<Step S240>

The MPU 130 selects the color temperature acquired for one camera thatis predetermined as a priority camera for which detection precision ofthe color temperature is considered to be higher among the camera 23Rand the camera 23L.

Here, FIGS. 14A and 14B are used to explain a phenomenon by which colortemperature detection precision varies, and a method to decide thepriority camera. FIG. 14A is a view that illustrates data created todesign the color temperature detection table of an image sensor (Ytype). Reference numerals 1401-1403 are curves that respectivelyillustrate fluctuation of the R average pixel value (average luminancevalue), the G average pixel value, and the B average pixel valueobtained in WB wave-detection (processing in step S180) by an individualcamera that has an image sensor (Y type) when the color temperature ofambient light is caused to change from 2500 through to 9500K. Here, whenanother camera equipped with a Y type image sensor is used to createsimilar data, characteristics do not bend up completely identical toFIG. 14A, and will be slightly shifted. This is an individualvariability of the camera.

FIG. 14B illustrates the color temperature detection table 801, wherethe WB ratio is made to be the abscissa axis, and the color temperatureis made to be the ordinate axis, and is created from data of FIG. 14Athat is created for each of a plurality of cameras (for each of whichthe equipped image sensor is the Y type) to consider individualvariability of a camera. For example, if the R and B average pixelvalues for all the cameras are calculated from the R and B average pixelvalues of each camera, and the WB ratio (the B average pixel value forall the cameras/the R average pixel value for all the cameras) iscalculated for each color temperature, it is possible to create thecolor temperature detection table 801. Of course, for a method ofcreating the color temperature detection table 801, a method other thanthis can be considered. Error bars in the figure illustrate ranges ofminimum and maximum variation of WB ratios of the measured data.

In the present embodiment, when acquiring the color temperature for eachof the cameras 23R and 23L, the color temperature detection table 801 isused. At this point, because the WB ratio varies in the range of theerror bars for the cameras 23R and 23L, the difference in colortemperatures acquired for the respective cameras is at its worst in acase of a camera combination having variation in which the colortemperature is the maximum for the camera 23R and the minimum for thecamera 23L, for example. Note that, performing measurements for allnumbers at a time of camera assembly and setting a color temperaturedetection table that is correctly adjusted for each individual may beconsidered, but because measurement allocating color temperatures takesmany man-hours, measurement for all numbers at the time of assembly isnot realistic as a method to fundamentally resolve this problem.However, measuring with only a specific color temperature of ambientlight at a time of camera assembly, for example, under a singlecondition for the standard color temperature 5500K, is realistic.Accordingly, it is possible to estimate to a certain extent what amountin a +/− direction the measure assembled camera is shifted from thecolor temperature detection table 801 which is representative.

FIG. 15 overlaps on FIG. 14B, an accurate color temperature detectiontable 1501 of the camera 23R for which the entire range of colortemperatures is allocated, and an accurate color temperature detectiontable 1502 of the camera 23L for which the entire range of colortemperatures is allocated. As described above, although generating thecolor temperature detection tables 1501 and 1502 allocating across theentire range of color temperatures is a serious effort, obtaining the WBratio at the standard color temperature 5500K of the present example isnot as cumbersome in comparison to generating the color temperaturedetection tables 1501 and 1502 allocating color temperatures across theentire range. Accordingly, if the standard color temperature is 5500K,the WB ratio for each of the cameras 23R and 23L at the colortemperature=5500K is obtained, and a camera for which a WB ratio closestto the WB ratio corresponding to 5500K in the color temperaturedetection table 801 is obtained is decided as the aforementionedpriority camera. In the case of FIG. 15, because the WB ratiocorresponding to 5500K in the color temperature detection table 1501 iscloser to the WB ratio corresponding to 5500K in the color temperaturedetection table 801 than the WB ratio corresponding to 5500K in thecolor temperature detection table 1502, the camera 23R is decided as theaforementioned priority camera.

Note that the step to decide the priority camera from the cameras 23Rand 23L explained above is performed before shipment of the head-mounteddisplay 200, and information indicating the decided priority camera isregistered in the memory 131 in the head-mounted display 200.

In addition, the above explanation was made in the case where thestandard color temperature is 5500K, and if the color temperature usedas the standard color temperature changes, the target decided as thepriority camera can change. However, considering the use environment ofthe head-mounted display 200, if it is decided in advance what colortemperature to set as the standard color temperature, it is possible todecide a camera that is desirable according to that to be the prioritycamera.

In addition, instead of simply setting the camera that is closest to theWB ratio in the color temperature detection table 801 as the prioritycamera for the standard color temperature, configuration may be taken todecide, as the priority camera, a camera that is closest to the WB ratioof the color temperature detection table 801 on average in colortemperatures in a neighborhood of the standard color temperature orthereabove. In other words, if a color temperature in the useenvironment of the head-mounted display 200 is envisioned, if it ispossible to decide, as the priority camera, a camera for which a WBratio that is desirable at the envisaged color temperature (in the aboveexample, closer to the color temperature detection table 801) can beobtained, various methods can be considered as the method to decidethat.

With this, even with a camera for which a shift from the colortemperature detection table 801 which is representative is large, bycombining it with a better camera to assemble the head-mounted display200, it is possible to achieve WB control that is comparatively stable.

In step S150, the MPU 130 sets the color temperature decided in stepS130 or the color temperature selected in step S240 as the colortemperature that is common (common color temperature) between thecameras 23R and 23L.

<Step S261>

The MPU 130, by performing processing in accordance with the flowchartof FIG. 7B, uses the common color temperature to obtain a WB correctionvalue to be used in WB correction on an image sensed by the camera 23R,and sends the obtained WB correction value to the camera 23R.

<Step S262>

The MPU 130, by performing processing in accordance with the flowchartof FIG. 7B, uses the common color temperature to obtain a WB correctionvalue to be used in WB correction on an image sensed by the camera 23L,and sends the obtained WB correction value to the camera 23L.

Thus, by virtue of the present embodiment, WB control for reducing WBcorrection error between cameras due to individual differences of imagesensors becomes possible.

First Variation

Although, in the second embodiment, the head-mounted display 200 hasimage sensors of a type that is the same, if the head-mounted display200 has image sensors of a type that is the same and has image sensorsof different types, the first and second embodiments may be used incombination, such as by using WB control of the second embodimentbetween cameras having image sensors of the same type and using WBcontrol of the first embodiment between cameras having image sensors ofdifferent types.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

What is claimed is:
 1. An image processing apparatus comprising: one ormore processors; and one or more memories storing instructions which,when executed by the processor, cause the image processing apparatus to:acquire a color temperature of each of a first sensed image that issensed by a first image sensing device and a second sensed image that issensed by a second image sensing device different from the first imagesensing device; and adjust color information of the first and the secondimage sensing devices so that both of the first and the second imagesensing devices use a common color temperature which is acquired byperforming a calculation process using both of the first and secondsensed images; derive a position and orientation of the first imagesensing device; and compose a virtual object, which is generated basedon the derived position and orientation of the first image sensingdevice, with the first sensed image.
 2. The image processing apparatusaccording to claim 1, wherein the adjustment of the color informationincludes adjusting each of R, G, and B signals in an image sensed by thefirst image sensing device and an image sensed by the second imagesensing device based on the common color temperature.
 3. The imageprocessing apparatus according to claim 1, wherein the instructions,when executed by the one or more processors, further cause the imageprocessing apparatus to obtain a difference between a maximum colortemperature and a minimum color temperature among the color temperaturesacquired from the first sensed image and the second sensed image.
 4. Theimage processing apparatus according to claim 1, wherein the acquisitionof the color temperature of each of the first sensed image and thesecond sensed image includes: obtaining an average pixel value for eachcolor component of the respective sensed images, and acquiring the colortemperature of each of the respective sensed images using the averagepixel value for each color component of the respective sensed images. 5.The image processing apparatus according to claim 4, wherein theacquisition of the color temperature of each of the first sensed imageand the second sensed image includes: obtaining the average pixel valueof an R component and an average pixel value of a B component of therespective sensed images, and acquiring a color temperature thatcorresponds to (the average pixel value of the B component)/(the averagepixel value of the R component).
 6. The image processing apparatusaccording to claim 1, wherein the image processing apparatus is ahead-mounted display equipped with the first image sensing device, thesecond image sensing device, and a display apparatus.
 7. The imageprocessing apparatus according to claim 1, wherein the derivation of theposition and orientation of the first image sensing device is performedbased on the second sensed image.
 8. The image processing apparatusaccording to claim 1, wherein a type of a sensor of the first imagesensing device differs from a type of a sensor of the second imagesensing device.
 9. The image processing apparatus according to claim 8,wherein the sensor of the first image sensing device is of a rollingshutter method, and the sensor of the second image sensing device is ofa global shutter method.
 10. The image processing apparatus according toclaim 1, wherein the adjustment of the color information includescalculating the common color temperature when a difference between thecolor temperatures acquired from the first and second sensed images issmaller than a predetermined value.
 11. The image processing apparatusaccording to claim 1, wherein the adjustment of the color informationincludes selecting one of the color temperatures acquired from the firstand second sensed images as the common color temperature when adifference between the color temperatures acquired from the first andsecond sensed images is greater than a predetermined value.
 12. A methodfor controlling an image processing apparatus, the method comprising:acquiring a color temperature of each of a first sensed image that issensed by a first image sensing device and a second sensed image that issensed by a second image sensing device different from the first imagesensing device; and adjusting color information of the first and thesecond image sensing devices so that both of the first and the secondimage sensing devices use a common color temperature which is acquiredby performing a calculation process using both of the first and secondsensed images; deriving a position and orientation of the first imagesensing device; and composing a virtual object, which is generated basedon the derived position and orientation of the first image sensingdevice, with the first sensed image.
 13. A non-transitory computerreadable storage medium storing a program which, when executed by acomputer comprising a processor and a memory, causes the computer toexecute a method comprising: acquiring a color temperature of each of afirst sensed image that is sensed by a first image sensing device and asecond sensed image that is sensed by a second image sensing devicedifferent from the first image sensing device; and adjusting colorinformation of the first and the second image sensing devices so thatboth of the first and the second image sensing devices use a commoncolor temperature which is acquired by performing a calculation processusing both of the first and second sensed images; deriving a positionand orientation of the first image sensing device; and composing avirtual object, which is generated based on the derived position andorientation of the first image sensing device, with the first sensedimage.